Variance estimation for NAEP data using a comprehensive resampling-based approach: An application of cognitive diagnostic models

This article presents an application of the jackknifing re-sampling approach (Efron, 1982) to error variance estimation for ability distributions of groups of students, using a multidimensional discrete model for item response data. The data utilized to examine the approach came from the National Assessment of Educational Progress (NAEP). In contrast to the operational approach used in NAEP, where plausible values are generated using the complete sample and are then subjected to a resampling scheme, the proposed approach re-estimated all model parameters for each of the replicate samples during the jackknife. The resampling approach proposed here is therefore a more comprehensive one because of its expected ability to represent the uncertainty due to sampling more appropriately. Results for the comprehensive resampling and re-estimation-based standard errors are presented for estimates of group means, total means, and other statistics used in NAEP for official reporting. Differences in results between the proposed approach and the operational approach are discussed.